Model selection for the rate problem: A comparison of significance testing, Bayesian, and minimum description length statistical inference

One particularly useful but under-explored area for applying model selection in psychology is in basic data analysis. Many problems of deciding whether data have ‘‘significant differences’’ can profitably be viewed as model selection problems. We consider significance testing, Bayesian and minimum description length (MDL) model selection on a common data analysis problem known as the rate problem. In the rate problem, the question is whether or not the underlying rate of some phenomenon is the same in two populations, based on finite samples from each population that count the number of ‘‘successes’’ from the total number of observations. We develop optimal Bayesian and MDL statistical criteria for making this decision, and compare their performance to the standard significance testing approach. A series of Monte-Carlo evaluations, using different realistic assumptions about the availability of data in rate problems, show that the Bayesian and MDL criteria perform extremely similarly, and perform at least as well as the significance testing approach. r 2005 Elsevier Inc. All rights reserved.